Never, EVER, assume the other driver knows what they're doing. Always anticipate the unexpected.
Having said that, I use left and right signal 99.999% of the time. For the 0.001%...that's when I'm on the road and it's vacant or there's no one within 15+ car lengths.